1. Field of the Present Invention
The invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image capturing apparatus, an image processing method, and a program capable of generating a high quality output image having a wide dynamic range by an image combining process using a plurality of images having different exposure times.
2. Description of the Related Art
A solid-state imaging device such a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor used for a video camera, a digital still camera, or the like performs photoelectric conversion of storing charges according to an incident light amount and outputting an electrical signal corresponding to the accumulated charges. However, since there is an upper limit to the charge accumulation amount in a photoelectric conversion element, if a predetermined light amount or more is received, an accumulated charge amount reaches a saturated level so that a so-call white-out where the saturated luminance level is set occurs in a subject area of which the brightness is a predetermined value or more.
In order to prevent this phenomenon, there has been a process where an exposure time is adjusted by controlling a charge accumulation period in the photoelectric conversion element according to an external light change or the like, so that sensitivity is controlled to an optimum value. For example, with respect to a bright subject, the charge accumulation period in the photoelectric conversion element is reduced by shortening the exposure time by a high speed shutter release, so that the electrical signal is output before the accumulated charge amount reaches the saturated level. By such a process, an image where the grayscale according to the subject is accurately reproduced can be output.
However, in the photographing of a subject where bright and dark portions are mixed, if the high speed shutter release is used, a sufficient exposure time is not secured for the dark portion. Therefore, the S/N is decreased, so that the image quality deteriorates. In this way, in the image photographing of the subject where the bright and dark portions are mixed, in order to accurately reproduce the luminance levels of the bright and dark portions, for a pixel of an image sensor where the incident light amount is small, high S/N may have to be implemented by a long exposure time, and for a pixel where the incident light amount is large, a process of avoiding the saturation may have to be provided.
As a method of implementing such a process, there has been known a method of using a plurality of images having different exposure times. In other words, in the method, a long time exposure image is used for a dark image area, and a short time exposure image is used for a bright image area which is whited out in the long time exposure image, so that an optimum pixel level can be determined. In this manner, by combining a plurality of images having different exposures, an image having a wide dynamic range, where there is no white-out, can be obtained.
For example, Japanese Unexamined Patent Application Publication No. 2008-99158 discloses a configuration of obtaining an image having a wide dynamic range by combining a plurality of images having different exposure amounts. The process is described with reference to FIG. 1. For example, in the moving picture photographing, an imaging device outputs two image data having different exposure times within a video rate (30-60 fps). In addition, in the still picture photographing, the imaging device also generates and outputs two image data having different exposure times. FIG. 1 is a diagram illustrating characteristics of the two images (the long time exposure image and the short time exposure image) having different exposure times generated by the imaging device. The horizontal axis is a time (t), and the vertical axis is an accumulated charge amount (e) in a light sensing photodiode (PD) constituting a photoelectric conversion element corresponding to one pixel of a solid-state imaging device.
For example, in the case where a sensed light amount of the light sensing photodiode (PD) is large, that is, in the case corresponding to a bright subject, as illustrated in a high luminance area 11 in FIG. 1, the charge accumulation amount is rapidly increased as time elapses. On the other hand, in the case where the sensed light amount of the light sensing photodiode (PD) is small, that is, in the case corresponding to a dark subject, as illustrated in a low luminance area 12 of FIG. 1, the charge accumulation amount is gradually increased as time elapses.
The times t0 to t3 correspond to the exposure time TL for obtaining the long time exposure image. With respect to the line illustrated in a low luminance area 12 where the exposure time TL is set to a long time, at the time t3, the charge accumulation amount does not reach a saturated level (unsaturated point Py). Therefore, an accurate grayscale expression can be obtained by the grayscale level of the pixel which is determined by using an electrical signal obtained based on the charge accumulation amount (Sa).
However, with respect to the line illustrated in a high luminance area 11, it is apparent that, before the time t3, the charge accumulation amount already reaches a saturated level (saturated point Px). Therefore, in such a high luminance area 11, only the pixel value corresponding to the electrical signal having the saturated level is obtained from the long time exposure image, so that the whited-out pixel is formed.
Therefore, in such a high luminance area 11, in a time before the time t3, for example, in the time t1 (charge sweeping starting point P1) illustrated in the figure, the accumulated charges in the light sensing photodiode (PD) are swept out. The charge sweeping is not performed on all the charges accumulated in the light sensing photodiode (PD), but it is performed to the intermediate voltage sustaining level controlled in the photodiode (PD). After the charge sweeping process, the short time exposure is performed again with the exposure time TS (t2 to t3). In other words, the short time exposure is performed during the period from the short time exposure starting point P2 to the short time exposure ending point P3 illustrated in the figure. The charge accumulation amount (Sb) is obtained by the short time exposure, and the grayscale level of the pixel is determined based on the electrical signal, which is obtained based on the charge accumulation amount (Sb).
In addition, in the case where the pixel value is determined based on the electrical signal, which is obtained based on the charge accumulation amount (Sa) obtained by the long time exposure in the low luminance area 12, and the electrical signal, which is obtained based on the charge accumulation amount (Sb) obtained by the short time exposure in the high luminance area 251, an estimated charge accumulation amount of the case of performing exposure for the same time or an electrical signal output value corresponding to the estimated charge accumulation amount is calculated, and the pixel value level is determined based on the calculated result.
In this manner, by combining the short time exposure image and the long time exposure image, an image having a wide dynamic range, where there is no white-out, can be obtained.
However, the plurality of the images having different exposure amounts is the images that are photographed at the timings different in terms of time. Therefore, if a motion of the subject occurs during the time, there are differences in the images at the time of combining the images. As a result, occurrence of pseudo color in an image area of the moving subject area results in a problem of deterioration in the image quality or the like.
As a related art disclosing a technology for reducing such a problem, there is, for example, Japanese Unexamined Patent Application Publication No. 2000-50151.
In Japanese Unexamined Patent Application Publication No. 2000-50151, disclosed is a configuration of comparing a plurality of images having different exposure amounts to specify a pixel area where there is a motion and performing correction. The details of the process are as follows. First, a long time exposure image (LE) and a short time exposure image (SE) are acquired, and an exposure ratio A (=LE/SE), which is a ratio of the exposure amount of the long time exposure image (LE) to the exposure amount of the short time exposure image (SE) is obtained. Furthermore, with respect to each pixel, (LE−SE×A) is calculated. In the case where the completely identical subject is photographed, (LE−SE×A)=0. With respect to a pixel where (LE−SE×A)=0 is not satisfied, there is a high possibility that different subjects are photographed in the long time exposure image (LE) and the short time exposure image (SE), and such a pixel area is identified as a motion area.
In other words, in the case where the same subject is photographed, a correspondence relationship between the output value (luminance) of the long time exposure image (LE) and the output value (luminance) of the short time exposure image (SE) is set on the line of a slope A (=exposure ratio) as illustrated in FIG. 2. In the case where the correspondence relationship deviates from the line, it is determined that there is a motion in the area.
However, in an actual case, since there is a variation in the special products of a PD or a transistor constituting the imaging device, it is difficult to accurately determine based on the satisfaction or otherwise of (LE−SE×A)=0 whether or not there is motion of the subject. For example, a little variation between the devices can be reduced by setting a threshold value (Th) and determining whether or not |LE−SE×A|<Th is satisfied. However, since the variation between the devices is varied according to the apparatuses, there is a problem in that it is difficult to set an optimum threshold value.